Metrics

Argo CD exposes different sets of Prometheus metrics per server.

Application Controller Metrics

Metrics about applications. Scraped at the argocd-metrics:8082/metrics endpoint.

MetricTypeDescription
argocd_app_infogaugeInformation about Applications. It contains labels such as sync_status and health_status that reflect the application state in Argo CD.
argocd_app_k8s_request_totalcounterNumber of Kubernetes requests executed during application reconciliation
argocd_app_labelsgaugeArgo Application labels converted to Prometheus labels. Disabled by default. See section below about how to enable it.
argocd_app_reconcilehistogramApplication reconciliation performance in seconds.
argocd_app_sync_totalcounterCounter for application sync history
argocd_cluster_api_resource_objectsgaugeNumber of k8s resource objects in the cache.
argocd_cluster_api_resourcesgaugeNumber of monitored Kubernetes API resources.
argocd_cluster_cache_age_secondsgaugeCluster cache age in seconds.
argocd_cluster_connection_statusgaugeThe k8s cluster current connection status.
argocd_cluster_events_totalcounterNumber of processes k8s resource events.
argocd_cluster_infogaugeInformation about cluster.
argocd_kubectl_exec_pendinggaugeNumber of pending kubectl executions
argocd_kubectl_exec_totalcounterNumber of kubectl executions
argocd_redis_request_durationhistogramRedis requests duration.
argocd_redis_request_totalcounterNumber of redis requests executed during application reconciliation

If you use Argo CD with many application and project creation and deletion, the metrics page will keep in cache your application and project’s history. If you are having issues because of a large number of metrics cardinality due to deleted resources, you can schedule a metrics reset to clean the history with an application controller flag. Example: --metrics-cache-expiration="24h0m0s".

Exposing Application labels as Prometheus metrics

There are use-cases where Argo CD Applications contain labels that are desired to be exposed as Prometheus metrics. Some examples are:

  • Having the team name as a label to allow routing alerts to specific receivers
  • Creating dashboards broken down by business units

As the Application labels are specific to each company, this feature is disabled by default. To enable it, add the --metrics-application-labels flag to the Argo CD application controller.

The example below will expose the Argo CD Application labels team-name and business-unit to Prometheus:

  1. containers:
  2. - command:
  3.   - argocd-application-controller
  4.   - --metrics-application-labels
  5.   - team-name
  6.   - --metrics-application-labels
  7.   - business-unit

In this case, the metric would look like:

  1. # TYPE argocd_app_labels gauge
  2. argocd_app_labels{label_business_unit="bu-id-1",label_team_name="my-team",name="my-app-1",namespace="argocd",project="important-project"} 1
  3. argocd_app_labels{label_business_unit="bu-id-1",label_team_name="my-team",name="my-app-2",namespace="argocd",project="important-project"} 1
  4. argocd_app_labels{label_business_unit="bu-id-2",label_team_name="another-team",name="my-app-3",namespace="argocd",project="important-project"} 1

API Server Metrics

Metrics about API Server API request and response activity (request totals, response codes, etc…). Scraped at the argocd-server-metrics:8083/metrics endpoint.

MetricTypeDescription
argocd_redis_request_durationhistogramRedis requests duration.
argocd_redis_request_totalcounterNumber of Kubernetes requests executed during application reconciliation.
grpc_server_handled_totalcounterTotal number of RPCs completed on the server, regardless of success or failure.
grpc_server_msg_sent_totalcounterTotal number of gRPC stream messages sent by the server.
argocd_proxy_extension_request_totalcounterNumber of requests sent to the configured proxy extensions.
argocd_proxy_extension_request_duration_secondshistogramRequest duration in seconds between the Argo CD API server and the proxy extension backend.

Repo Server Metrics

Metrics about the Repo Server. Scraped at the argocd-repo-server:8084/metrics endpoint.

MetricTypeDescription
argocd_git_request_duration_secondshistogramGit requests duration seconds.
argocd_git_request_totalcounterNumber of git requests performed by repo server
argocd_git_fetch_fail_totalcounterNumber of git fetch requests failures by repo server
argocd_redis_request_duration_secondshistogramRedis requests duration seconds.
argocd_redis_request_totalcounterNumber of Kubernetes requests executed during application reconciliation.
argocd_repo_pending_request_totalgaugeNumber of pending requests requiring repository lock

Prometheus Operator

If using Prometheus Operator, the following ServiceMonitor example manifests can be used. Add a namespace where Argo CD is installed and change metadata.labels.release to the name of label selected by your Prometheus.

  1. apiVersion: monitoring.coreos.com/v1
  2. kind: ServiceMonitor
  3. metadata:
  4.   name: argocd-metrics
  5.   labels:
  6.     release: prometheus-operator
  7. spec:
  8.   selector:
  9.     matchLabels:
  10.       app.kubernetes.io/name: argocd-metrics
  11.   endpoints:
  12.   - port: metrics
  1. apiVersion: monitoring.coreos.com/v1
  2. kind: ServiceMonitor
  3. metadata:
  4.   name: argocd-server-metrics
  5.   labels:
  6.     release: prometheus-operator
  7. spec:
  8.   selector:
  9.     matchLabels:
  10.       app.kubernetes.io/name: argocd-server-metrics
  11.   endpoints:
  12.   - port: metrics
  1. apiVersion: monitoring.coreos.com/v1
  2. kind: ServiceMonitor
  3. metadata:
  4.   name: argocd-repo-server-metrics
  5.   labels:
  6.     release: prometheus-operator
  7. spec:
  8.   selector:
  9.     matchLabels:
  10.       app.kubernetes.io/name: argocd-repo-server
  11.   endpoints:
  12.   - port: metrics
  1. apiVersion: monitoring.coreos.com/v1
  2. kind: ServiceMonitor
  3. metadata:
  4.   name: argocd-applicationset-controller-metrics
  5.   labels:
  6.     release: prometheus-operator
  7. spec:
  8.   selector:
  9.     matchLabels:
  10.       app.kubernetes.io/name: argocd-applicationset-controller
  11.   endpoints:
  12.   - port: metrics
  1. apiVersion: monitoring.coreos.com/v1
  2. kind: ServiceMonitor
  3. metadata:
  4.   name: argocd-dex-server
  5.   labels:
  6.     release: prometheus-operator
  7. spec:
  8.   selector:
  9.     matchLabels:
  10.       app.kubernetes.io/name: argocd-dex-server
  11.   endpoints:
  12.     - port: metrics
  1. apiVersion: monitoring.coreos.com/v1
  2. kind: ServiceMonitor
  3. metadata:
  4.   name: argocd-redis-haproxy-metrics
  5.   labels:
  6.     release: prometheus-operator
  7. spec:
  8.   selector:
  9.     matchLabels:
  10.       app.kubernetes.io/name: argocd-redis-ha-haproxy
  11.   endpoints:
  12.   - port: http-exporter-port

For notifications controller, you need to additionally add following:

  1. apiVersion: monitoring.coreos.com/v1
  2. kind: ServiceMonitor
  3. metadata:
  4.   name: argocd-notifications-controller
  5.   labels:
  6.     release: prometheus-operator
  7. spec:
  8.   selector:
  9.     matchLabels:
  10.       app.kubernetes.io/name: argocd-notifications-controller-metrics
  11.   endpoints:
  12.     - port: metrics

Dashboards

You can find an example Grafana dashboard here or check demo instance dashboard.

dashboard